Papers with dropout mechanism
SelfMix: Robust Learning against Textual Label Noise with Self-Mixup Training (2022.coling-1)
Copied to clipboard
| Challenge: | Existing methods to handle label noise in text classification tasks are limited to visual data. |
| Approach: | They propose a method to handle label noise in text classification tasks using a Gaussian Mixture Model. |
| Outcome: | The proposed method outperforms baselines on three types of text classification tasks on visual and textual data. |
Scheduled DropHead: A Regularization Method for Transformer Models (2020.findings-emnlp)
Copied to clipboard
| Challenge: | DropHead is a structured dropout method for regularizing multi-head attention . DropHed drops entire attention heads during training to prevent overfitting . |
| Approach: | They propose a structured dropout method specifically designed for regularizing multi-head attention mechanism . DropHead drops entire attention heads during training to prevent overfitting . |
| Outcome: | The proposed method can improve transformer models by 0.9 BLEU score on translation task and around 1.0 accuracy for various text classification tasks. |
SDA: Simple Discrete Augmentation for Contrastive Sentence Representation Learning (2024.lrec-main)
Copied to clipboard
| Challenge: | Existing methods for data augmentation have not been well explored. |
| Approach: | They propose to use punctuation insertion, modal verbs, and double negation to produce diverse forms of sentences. |
| Outcome: | The proposed methods perform better on diverse datasets with semantic similarity and standard negation. |